Spatio-temporal layout of human actions for improved bag-of-words action detection

5 We investigate how human action recognition can be improved by considering spatio-temporal layout of 6 actions. From literature, we adopt a pipeline consisting of STIP features, a random forest to quantize the 7 features into histograms, and an SVM classifier. Our goal is to detect 48 human actions, ranging from simple 8 actions such as walk to complex actions such as exchange. Our contribution to improve the performance of this 9 pipeline by exploiting a novel spatio-temporal layout of the 48 actions. Here each STIP feature does not in the 10 video contributes to the histogram bins by a unity value, but rather by a weight given by its spatio-temporal 11 probability. We propose 6 configurations of spatio-temporal layout, where the varied parameters are the 12 coordinate system and the modeling of the action and its context. Our model of layout does not change any 13 other parameter of the pipeline, it requires no re-learning of the random forest, yields a limited increase of the 14 size of its resulting representation by only a factor two, and at a minimal additional computational cost of only 15 a handful of operations per feature. Extensive experiments show that the layout is demonstrated to be 16 distinctive of actions that involve trajectories, (dis)appearance, kinematics, and interactions. The visualization 17 of each action’s layout illustrates that our approach is indeed able to model spatio-temporal patterns of each 18 action. Each layout is experimentally shown to be optimal for a specific set of actions. Generally, the context 19 has more effect than the choice of coordinate system. The most impressive improvements are achieved for 20 complex actions involving items. For 43 out of 48 human actions, the performance is better or equal when 21 spatio-temporal layout is included. In addition, we show our method outperforms state-of-the-art for the 22 IXMAS and UT-Interaction datasets. 23 24